[SE-4481] Feature to upload edX service logs - #6418

Thanks for the pull request, @Skchoudhary! I've created OSPR-5797 to keep track of it in JIRA, where we prioritize reviews. Please note that it may take us up to several weeks or months to complete a review and merge your PR.

Feel free to add as much of the following information to the ticket:

• supporting documentation
• Open edX discussion forum threads
• timeline information ("this must be merged by XX date", and why that is)
• partner information ("this is a course on edx.org")
• any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.

Please let us know once your PR is ready for our review and all tests are green.

@Skchoudhary Thank you for your contribution. Please let me know once it is ready for our review.

@Skchoudhary Please note that we cannot proceed with reviews for any of your PRs until we receive an answer from you to my email, sent on May 24. I will re-send it you right now, please check your spam folder and reply as soon as possible.

Hi @natabene, I have replied to your email.

@Skchoudhary Thank you, you are all set now.

Hi @natabene
This PR is ready for edX's review

@Skchoudhary 🎉 Your pull request was merged!

Please take a moment to answer a two question survey so we can improve your experience in the future.

@Skchoudhary unfortunately we had to revert this PR because it broke log upload and instance shutdown on the edx.org infrastructure. We'll need to figure out how to preserve the old behavior for edx.org while allowing the behavior you need. Probably need to use boolean flags.

@Skchoudhary It looks like the error we got was "msg": "'COMMON_OBJECT_STORE_LOG_SYNC_PREFIX' is undefined". So I think you need to make sure there's a sensible default for this variable. Also I think we just want to sync tracking logs, and not the lms and cms logs, because we only sync those via Splunk and don't need to sync them via Splunk, so we need syncing of those logs to be optional.

@jdmulloy Thanks for pointing that out, will only sync the tracking logs for now.

Hi @pkulkark,

Can you please have look into the issue as me might have to update the other PR also? If I got time this weekend, I can check this If you want.

No problem @Skchoudhary I'll take care of it.

@jdmulloy I opened a new PR #6465 that re-introduces the option to sync edx-services logs but only when the COMMON_OBJECT_STORE_EDX_LOG_SYNC flag is set to true. Could you review it?